A Linear Memory CTC-Based Algorithm for Text-to-Voice Alignment of Very Long Audio Recordings

Synchronisation of a voice recording with the corresponding text is common task in speech and music processing, used many practical applications (automatic subtitling, audio indexing, etc.). A approach derives mid-level feature from finds its alignment to by means maximizing similarity measure via Dynamic Time Warping (DTW). Recently, Connectionist Temporal Classification (CTC) was proposed that directly emits character probabilities uses those find optimal text-to-voice alignment. While this method yields promising results, memory complexity search remains quadratic input lengths, limiting application relatively short recordings. In work, we describe how recent improvements brought textbook DTW algorithm can be adapted CTC context achieve linear complexity. We then detail our overall solution demonstrate it align several hours mean error 50 ms for speech, 120 singing voice, which corresponds median below both types. Finally, evaluate robustness transcription errors different languages.